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Foreword 

„ The methodological review o{ the Children's English and Services Study (CESS) and a 
response to the review s estimation procedures discuss issues concealing the state of tfie art in 
language proficiency assessment in general and analytical issues regarding the Language 
Measurement and Assessment Inventory , developed for the CESS, in particular. Providing a 
single source for these documents, which are no longer available from the National Center 
for Education Statistics and the National Institute of Education, this volume is publ^hed in 
conjunction with two other reports that present results of the CESS: Language Minority 
Children with Limited English Proficiency in the United States and Educational Needs 
Assessment for Language Minority Children with Limited English Proficiency. Both of these 
reports, written by J. Michael O'Malley, are also available from the National Clearinghouse 
for Bilingual Education. 

A national investigation of the number and educational needs of language minority 
students with limited English proficiency and of the instructional services provided to them, 
the CESS represents the first time that a number of important methodological refinements 
over previous investigations converged. In addition to the methodojogical reviews, the 
present volume contains^ brief discussion of the study design iSSufes, sample representation, 
defining limited English proficiency, and determining a criterion cut-off score, followed by a 
summary of the major methodological issues. , %• 

One of the activities of the National Clearinghouse for Bilingual Education is to publish 
documents addressing the specific information needs of the bilingual education community. 
We are pleased to make this title available through our growing list of pubficatiops. 
Subsequent Clearinghouse products will similarly seek to contribute information that can 
assist in the education of minority language and culture groups in the United States. , 

National Clearinghouse 
for Bilingual Education 
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L Introduction 



As part of the 1974 (ESEA Title VII) bi- (NCES) and a reanalysis by the National 
lingual education legislation, Congress ban- Institute of Education (NIE/. They are, 
dated a count or estimate of the number of « respeetfrely, "The Children's Epglish arid 
children who were limited in English pro- '.'Services Study: A Methodological Review" 
ficiency (LEP) and a determination of their and ^Reanalysis of the Number of Limited 
basic educational needs. The Children's Eng- English Proficient Students Estimated .in the 
hsh and Services Study (CESS) of 1978 was Children's English and Services Study. * In 1 
the response to this mandate. The CEiSS was ordet to place the two reviews in the appro- 
cQnducted for the National Center for Edu- J priate context and to aid in understanding 
cation Statistics (NCES) and the National the methodological issues of the CESS pro- 
Institute of Education (NIE) by L. Miranda ject, the methodological reviews are preceded 
and Associates of Washington, D.C. as by a brief background* of the CESS and a 
primary contractor. 1 summary of the main methodological con- 
Information concerning the number of siderations. 
limited English proficient children is needed 
in order to establish projections concerning Daniel M. Ulibarri 
the appropriate levels of services and to Research Associate t 
determine the prQbable impact of bilingual National Institute of Education 
education. Because estimates of the number 4 * 
of LEP children are so important and 

because the CESS is the only national study Notes 
of its kind that ^attempts to provide such 

estimates, information concerning the method; , j Michael' O'Malley, Children s English 
ology of the study must be made available. , md Servkes Stud}>t Spring 197 g (Rosslyn, 
Additionally, the CESS is the only national Va . National clearinghouse for Bilingual 
study of the language proficiency of language Education, 1981). ' 
minority groups in which a language profi- 
ciency test was administered, and its results 2. Rebecca Oxford et al., Projections of 
have been used in subsequent studies, includ- Non-English Language Background and- 
ing one estimating the number of LpP chil- . Limited English Proficient Persons in the 



dren in projections to the year 2000. 2 



The purpose of the present d6cument is to 
give information on the methodological 
issues involved in the CESS. The volume 
consists of a review of the CESS by the 
National Center for Education Statistics 




cation Statistics (NCES) by InterAmerica 
Research Associates, Inc., Contract OE, 
300-79-0737 (Rosslyn, Va.: InterAmerica 
Research-Associates, 1980). 
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U. The Childress English and 
Services StudV(CESS) 



The Children's English and Services Study 
(CESS) was a direct result of a Congressional 
mandate as stated m the Bilingual Education 
, Act, 'as amended 'in 1974 (Title VII of the 
Elementary and ^Secondary Act). The pur- 
pose of the CESS was to determine the 
number of children of limited English profi- 
ciency (LEP) aged* 5 to 14 living in house- 
holds where a language other than English 
was spoken. In addition, the CESS was 
" designed to provide estimates of the number 
of limited English speaking children Jrom 
Spanish language backgrounds and the ag- 
gregate of all other language minorities com* 
bincd in major geographic regions of the Uni- 
ted States. The results indicate that in Spring 
1978 there were 2.4 million LEP children in 
the 5 to 14 age range and an estimated 3.6 
million in the U.S. school-age population (4 • 
to 18 years old). For the age-range 5. to 14 
years there were 1.7 million Hispanic LEP 
children and .7 million LEP children* in, all 
other language minority backgrounds com- 
bined. ^ 

Study Design Issues < 

Since the Congressional mandate for a 
count of limited EngliftTspealcing children 
called for information that no previous inves- 
tigation had attempted to produce at the 
national level, a new study design had to be 
developed. The m^pr problems this design 
had to d^al with wc|b: 

• Identifying a representative sample. v\ ' 
procedure was retired for developing 
a probability sample of the population 
of language minoritj|j>ersons. 

• Defining limited Englkh^speaking abil- 
ity. Acceptable criteriaW test instru- 
ments for identifying limited English 
sneaking Children as defined in ESEA 
Title VII fyerc required tliat could be 
Msed in a household survey. V 

• Determining ^criterion scoreJAccctft- 
able techniques^were needed fotidenti- 
fying a criterioiff score below wliich all 



students could be accurately classified 
as limited English speaking. 

Sample Representation 

The CESS was based on a sample of 
households in the United States in which* a 
language other than English w^s spoken usu- 
ally or often. Adults were interviewed in the 
Spring of 1978 In a randomly selected, 
nationally representative probability sample 
of 35,000 households. Approximately 2,200 
households amohg these 35,000 were identi- 
fied where a language other than English was 
spoken and where children between the ages 
of 5 and 14 were living. Within these 2,200 
households, selected children (up to 2 chil- 
dren per household) were individually admin- 
istered a test in 'English that determined 
whether or not they were limited in English 
language proficiency. The sample was de- 
signed to be representative of the number of 
children in California, Texas, New York, arid 
the remainder of the country. 

Defining Limited English Proficiency 

The external criterion that served as the 
Operational definition of limited English pro- 
ficiency and that was used to measure Eng- 
lish proficiency was a specially constructed 
language assessment instrument. Specific 
needs and constraints that guided the devel- 
opment of an accepta>le criterion to be used 
for defining limited English proficiency were 
the following: 

1. The definition of limited English speak- 
ing-ability had to be consistent with the 
Title VII legislative definition stated in 
the Bilingual Education Act, as amended 
in 1974. The legislation pointed to two 
requirements for defining' limited Eng- 
lish proficiency: non-English language 
' background, defined in terms of native , 
language, country of origin, or language 
• envirQnment, and limited ability in 
Speaking and understanding English tp 
the extent of not being able to profit 
from instruction in English. 
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2. The language proficiency criterion had. 
to produce a dichotomous classification*., 
of limited versus not limited English- 

* speaking ability. 

3. A criterion had to be, adopted that 
would be usable to define limited Eng- • 
.lish speaking ability in the context of a 
household survey for children aged 5-14 
years. / 

Since no existing instrument met the needs 
and constraints of the ?tudy or satisfactcfrfy 
identified limited English proficiency con- 
sistent with the ESEA Title VII definition, a 
new instrument was developed, the Language 
Measurement and Assessment Inventory 
(LM&AI). (A summary is provided as Ap- 
pendix A to the second methodological 
review in this volume.) Jho test measures 
age-specific speaking, listening, reading, and* 
writing skills in English, and was designed to 
meet the definition of limited English profi- 
ciency in the Bilingual Education Act. Repre- ^ 
sentatives from 30 State Education Agencies 
developed the test specifications <ind served 
on a review team for the study. 

The LM&AI was developed to different 
tiate language minorities who were limited in 
English from those who could p^fit from 
instruction in English. In developing £he 
instruments, limited English proficient chil- 
dren and fluent English speaking children 
who were benefiting from instruction in Eng- 
lish were identified by local school personnel 
" as the criterion groups. 

The test criterion for limited English profi- 
ciency is a cut-off point on the total score 
that most accurately classifies children as 
limited in English for their age level. Children 
in the national sample whose score fell below 
the critical value were classified as limited 
English proficient anti those* whose score fell 
above the critical value were classified as flu- 



ent English proficient. The criterion' for 
determining a cut-off score was derived from 
a field study described briefly in the following 
section. 



Determining a Criterion Cut-off Score 

In selecting an approach to identify a cut- 
off score appropriate for each age group, an . 
attempt was 'made to maximize the likeli- 
hood of coriracf classifications of children 
who were limited and those who were fluent 
in English proficiency. The approach selected 
was discriminant function analysis. Discrim- 
inant function analysis is similar methodolog- 
ically to regression analysis when the criteri- / 
on variable is dichotomous. Thus, in dis- 
criminant function analysis orite or more 
variables are used to predict group member- 
ship in one of two groups (in this case LEP 
and non-LEP). In the CESS, the predictor 
was the continuous score on the English lan- 
guage instrument and the .criterion was the 
school's designation as limited or fluent in 
English speaking ability. The discriminant 
function maximizes the overall number of 
accurate classifications such that correctly 
classifying fluent English speaking children 
♦was giyen as much weight as correctly classi-^ 
fying children who were limited in English 
proficiency. 

Cut-off^ scores were determined on an 
independent sampteof students obtained in a 
field test of the instrument. Accurate classifi- ' 
cations in predicting language ability occur 
when a child is classified by both the predic- 
tor and criterion as limited English profi-* 
cient, or as fluent in English. The ratio of the 
number of correct classifications to the total 
» number of cases yields an accuracy rate. For 
* the field test, the overall accuracy rate ranged 
from 82.9% to 97.2%, depending on the age 
, level of the children tested/ 



III. MajorMethodoIogical Issues 



In January 1980, the National Center for 
Education Statistics, Office of Research and 
Analysis (NCES/ OR A), U.S. Department of 
Education issued a report entitled "Analyti- 
cal Issues Regarding the Children's English 
and Services Study (Al/CESS)" (see Appen- 
dix A of NCES/ORA Methodological Re- 
view, p. 23 of this •volume). The Al/CESS 
contained a discussion of three analytical 
issues that were identified upon review of the 
CESS Draft Report of September 1979, and>> 
a later revision dated Noverqber 1979. In the 
A 1/ CESS report a 'call was made for 
responses to the issues (Appendix B to the 
NIE reanalysis contains L. Miranda and 
Associates 1 response to the Al/CESS report, 
pa^e 47 of this volume). As a result of the 
responses, NCES/ORA prepared ^ metho- 
dological review of the CESS in August 
1980. In the review, NCES OR A discussed 
three analytical issues: . 

v 

I. Were the items selected for inclusion in 
the^Language Measurement and Assess- 
ment Inventory (LM&A1) selected prop- 
erly? 



2. Were the cut-off scores for the LM&A1, 
which were determined and used to 
classify children as either English profi- 
cient or of limited English proficiency 
(LEP), set properly? 

3. What were the effects of non^response 
^ bi&s on the counts and estimates of the 
' number of LEP children? 

V 

In response, to the NCES/OR^A review, 
NlEpreparedB report comparing the results 
of the CESS methodology Tor determining 
cuf-off scores with the alternative procedure 
recommended in the NCES/ORA review. 
While the N.CES comments focused on three 
issues, the NIE response addressed only the 
second issue. 

The other two issues were adequately dis- 
cussed in the NCES document and further 
comment by NIE was not necessary. The fol- 
lowing section contains the NCES/ORA 
Methodological Review of the CESS report 
and NIE's response to the issue of determin- 
ing cut-off scores. 
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, Introduction 



On January" 21, 1980. the Office of 
Research and Analysis (OR A) of the National 
Center for Education Statistics (NCES), U.S. 
Department of Education- issued a report 
entitled l4 Analytical Issues Regarding the 
Children's English and Services Study" (Al, 
CESS). The purpose of the Al ; C^SS report 
was: 

to present and discuss three analytical issues 
which ha\e been identified as a result of a post 
hoc assessment of the research design, data anal- 
>ses. and other information yvhich are described 
^in the 1978. Children's English and Services 
Study (CESS) Draft Report of September 6. 
1979 (and a .later rcusion dated November 
1979) ! 

A copy of the AI/CESS report is found in 
Appendix A, page 23 of this volume. 

The objective of the present NCES OR A 
inquiry is best summarized by the following 
passage from the January 1980 report: 

Since the results of the 1978 CESS are of tre- 
mendous importance to, present and future 
research studies, bilingual [education] program 
and pohc\ development, and funding for bilin- 
gual education, unresolved analytical issues * 
which could adversely affect the validity of the 
results are being stated with the hope of their 
resolution* 2 

Secondary analyses of data and research 
designs frequently reveal analysis errors or 
areas of skepticism in the design. Sterling and 
Weinkam (1979). who discovered misclassifi- 
catibn in a study of mortalit^mong U.S. 
veterans, describe the potentiaFresptonse of 
managers to this discovery as either "cc 
ative" or "adversary." 

In the former case, management tries to 
determine the source concern arid to restruc- 
ture the procedures or analyses. In the latter 



case, the respopse may be to eliminate the 
discovery of errors rather than their source. 
Regarding this prQblem, 'Sterling and Wein- 
kam further observed that: 

. there may be underlying sociological and 
ps\ etiological forces operating which make it 
. \more acceptable for management to adopt an , 
adversary rather than a cooperative stance even 
iO scientific instances. From a simpleminded 
perspective, to acknowledge the existence of 
errors may require considerable efforts and 
expenditures to correct them, not to say anything 
of extracTing accountability from some individu- 
als- who insist on bringing these errors to the 
attention of management as troublemakers. 3 . 

They continued by-mdicating that "a$, the 
value of using available data files for second- 
ary analysis becomes increasingly clear (and * 
there is a great deal of value in the use of 
properly collected and suitable data), .we may^* 
expect that other discoveries similar to ours 
will be made. 1 * They recommended that a 
mechanism be established to encourage second- 
ary analysis for validating the appropriate- 
ness of preceding work and to distribute the 
results of these analyses., 

NCES/ OR A recognizes that many re- 
search reports are incomplete Qr inaccurate 
in one or more ways, and the CESS Draft 
Report mentioned earlier is no exception: 
Accordingly, NCES/ORA's purpose in this 
report is to clarify the issues and make spe- 
cific recommendations for modifications to 
the N1E Draft Report. , 

The preseri't report includes a discussion of * 
the three issues presented in the AI/CESS 
report and is based, in part, upon the 
responses, received from it. In addition, perti- 
nent literature on language development and 
assessment, the relationship of language 
acquisition to cognition, and additional data 
analyses completed by NCES provide the 
bases for this'paper. 
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ment of the Issues 



The following analytical issues are the sub- 
ject of this position paperf* *V 

L Were the items that we re selected for 
inclusion in the LarignsgTfvIeasuitment 
and Assessment xnventofy (LM&A1) 
selected properlyV; 



2. Were,thecut r off scores for the LM&Ad, 
which weri determined and used to 
classify children as either English profi- 
cient or of limited English proficiency 
(LEP), set properly? 

3. What were the effects of non-response 
bias on the counts and estimates of the 
number of LEP children? 



Discussion of the Issues 
and Recommendations 



Issue: ' ) Were the items that were 
selected for* inclusion^ in 
>r- the Language Measure- 

ment and Assessment In- ? 
ventory fLM&AI) selected 
properly? 

This issue was restated as: •» 

Is English language proficiency the dimen- 
1 sion on which the scores vary, or are other 
dimensions associated \vith variations, in 

9 

the scores? 
Two subissues were posed, namely: 

• Are the test scores related to language 
dominance? 

• Are the test scores related to general 
language development? 

Discussion 

Lourdes Mir^ida, President of L. Miranda 
and Associates, the prime contractor for the 
CESS, responded to the Al/CESS report. In 
discussing the rationale for the test items 
selected for the LM&AI, Miranda noted that 
"it was essential for jus to measure the ability 
[of language minority children] to successfully 
deal with academic classrpom tasks that are 
often as "clearly reliant on memory and cogni- 
tive abilities as on English language skills." 6 
Therefore, "other dimensions [e.g., cognitive] 
are associated ifoith Score Variation/' 7 



The LMt&AI was specifically designed to 
meet the definition of limited English profi 1 
ciency found in the Bilingual Education- Act, 
'the 1965 Etementary and Secondary Act 
(ESEA), Section 703 (a) (1) (B), as amended. 
The 1978 amendment of the act expandedthe 
language skill domains from "limited English 
speaking ability" to include speaking, read- 
ing, writing, and understanding'English. By 
virtue of their limited English proficiency, 
Congress concluded, language minority chil- 
dren were denied the opportunity to attain 
levels comparable to others at their appro- 
priate age and grade levels. 

J. Michael CTMalley, the NIE project 
officer for CESS, responded to the first issue 
that: 

Because functioning in the classroom often 
requires conceptual skills as well as oral language 
and literacy, the inclusion of cognitive demands, 
in the test items was seen as an acceptable 
approach for increasing the contcnt«and perhaps 
the predictive validity of the test. 8 

O'Malley also stated that: ~ 

A "pure" measure of English proficiency could 
not have possessed the content validity required 
to identify language minority children \vho have 
difficulty profiting from instruction in English 9 

Earlier in his response, O'Malley said, 
"simply stated, the, test scores are predictors 
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of the ability to profit from English language 
instruction, which determines eligibility for . 
ESEA Title VII" 10 He observed, "School 
decisions about eligibility for ESEA Title VII 
are often based on a child's general level of 
functioning in the classroom rather than on 
English language proficiency alone 11 .... 
The LM&AI used tested skills in Eng- 
lish . . . to simulate -the decisions schools 
would make in determining that language 
minority students could not profit from 
instruction in English. " ,2 

A reviewof recent literature in the areas of 
language assessment, linguistic and intellec- 
tual development, and bilingual education 
programs revealed that analytical questions 
in these areas Kave, for.some time, presented 
serious intellectual challenges to researchers 
and educators. The issues raised in the 
AI/CESS report were presented within the 
framework of the CESS development pro- 
cess and with the knowledge that there are 
many unanswered basic research questions in 
the; three areas mentioned above. NCESjs f 
purpose *n this report is to clarify current 
thought on these issues. 

The first subissue raised in the AI/CESS 
report was stated as: "Are the scores related 
to language dominance?" 13 O'Malley takes 
the following position; 

By exclusion in the [legislative] definition of eli- 
gibility, language dominance has no role in policy 
determination for ESEA Title VII eligibility. 
Thus, the statements in the NIE report on the- 
CESS that language dominance was considered 
irreievant4are]understarKlablc. J4 — — 

NCES/ORA believes there is justification 
for excluding a language dominance concept 
in the development of the CESS. Specifically, 
there does not appear to be agreement 
among linguists about an operational defini- 
tion and, therefore, about the impact of "lan- 
guage dominance" upon the ability of lan- 
guage '^minority children to profit from 
instruction in English. 

DeAvila and Duncan (1976) argue against 
using a "language dominance* 1 concept when 
discussing school achievement. They ask, 
"How does the concept of [language] domi- 
nance clarify the relation between the child's 
linguistic development and school achieve- 



ment in such a way that we can do something 
about it?" 15 They continue, "Another way of 
asking this question is, by asking whether pr 
not 'dominance * in or of itself determines what 
is learned or* what can be learned." 16 .Lan- 
guage dommance "does not address the real 
issue that the child might have language 
development problems in both Iaijguages — 
the native language and English,* 117 

Some experts have argued that a language 
dominance concept is meaningful only when 
the use of a language is considered within a 
social or cultural context, such as home and 
family relationships, social interactions, an 
academic domain, a business environment, 
or within a religious context. The degree of 
fluency or level of language dominance is 
meaningful onl> when the purposelor which 
language is being used is also stated. In this 
sense, several "dominance" levels might be 
defined. 

~~ Regarding a child's possible difficulty with 
both languages, Dubois (1980) States: 
"Whether it is appropriate to assess English 
language proficiency, [while] ignoring the 
child's proficiency in anpther language, re- 
mains $ policy question to be addressed.* 118 
More specifically, this is an empirical ques- 
tion. A recent article by Cummins (1979) 
addresses this question. In the following 
excerpt L, refers to a child's first language 
and Lj refers to the second language* Cum- 
mins says: 

The lack of concern for the developmental 
interrelationships between language and thought 
in4h^ilingua^chiW-»^eofihrmajorreasons — 
why evaluations and research have provided so 
little data on the dynamics of the bilingual child's 
interaction with [his or her] educational environ- 
ment. A direct determinant of the quality of this ^ 
interaction is clearly the level of Lj and L2 com- 
petence which the bilingual child develops over 
thecourse of [his or her] school career. . . . What 
level of L2 competence must the child possess at 
various grade levels in order to benefit optimally 
from instruction in that language? ... To what 
Extent are Lj and L2 skills interdependent and 
what are the implications of possible interdepen- 
dencies for cognitive and academic progress? In 
other words, do children who maintain, and 
develop their Li in school .develop, higher or_. 
lower L2 levels of skills than those whose L| is 
replaced by their L2? 19 



1 
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4 Cummins provides research evidence for a 
developmental interdependence hypothesis 
which says that the level of L : competence 
that a bilingual child attains is partly a func- 
tion of the type of competence the child has 
developed in L, at the time when intensive 
exposure to L 2 begins. In this sense, a mea- 
sure of L t proficiency is important for policy 

^decisions. ! 

Lambert (1975) suggests that children 
exhibit eit her "additive bilingual^m" or "sub- 
tractive bilingualism. 0 A child s bilinguahsm 
is most likely to be additive when Lj is presti- 
gious or the dominant language and is, there- 
fore, not in danger of being replaced by L : . In 
this case, a I5!lingual child adds L 2 skills with- 
out, losipg L, skills. Subtractive .bilinguahsm 
refers to ahe form children experience when 
their L, is eventually replaced by L : , and gen- 
erally occurs when the child's L, is a nonpres- 
tigious pi* minority language Socioeconomic 
status also seems to be a factor that is related 
to whether a child's bilinguahsm is subtrac- . 
tive or additive. Children from upper or mid- 
dle class socioeconomic strata," when given 
instruction in L :% tend' to experience additive 
bilingualism, while children from lower socio- 
economic status tend to experience subtrac- 
tive bilingualispi. Troike (1980) hypothesizes 
that, for children from lower socioeconomic 
groups, cognitive development ca^n become 
disrupted whetfi children begin learning* L 2 
between the a£es of 6-10. Socioeconomic sta- 
tus and socippolitical status are, therefore, 
-^late<^to-^ngtiagrand~rogffitiv^el^Iop^' 
ment. [ s 

The second subissue in the Al, CESS 
report was: n Are the test scores related to 
general language development?" 20 The con- 
cern was foV the inclusion of test items with 
cognitive components on the LM&^l. Mi- 
randa noted earlier that the purpose of the 
LM&Al was to measure the ability of lan- 
guage minority children to deal successfully 
with academic elassroom language skills. In 
reply to this-subissue, Miranda stated that ^*it 
is difficult to imagine how a test* of 'pure' 
^linguistic competence could have been devel- 
oped should we have been asked to do so." 21 
DeAvila et al. (1979) observe that "much 
confu^on abounds with respect 1o both the 



meaning and the measurement of English 
language proficienc)."-- Moreover, they note 
that "the role of language and cognition in 
general is itself not clearly agreed upon." 21 
For the purpose of this discussion, cognition 
means the act or process of perceiving or 
knowing. 

Cazden ( 1972) addresses two controversial 
items ,of interest in Child Language and 
Educatiern. The first concerns whether a per- 
son s thought 'is affected by the particular 
^ language forms or speech patterns with 
which he or she is familiar. The second item 
cdncerns the question of which dev elops first, 
the nonverbal idea or the words to express 
it. 24 Essentially, this poses the central issue: 
Which develops first, language or cognition? 
Language experts, educational psychologists, 
and professionals in related fields apparently 
do not agree upon the^roposed answers to 
this quest^n. 

Cazdens first item is based upon the 
Whorfian (1956) hypothesis that says* that 
"language influences our perceptions of and 
responses to the 'world." 25 Thisleads us to # 
believe that no learning can take place until 
language proficiency is attained; therefore, 
language determines cognition. Regarding 
Cazdens Second item, Piaget indicates a 
child s cognitive development is the primary 
factor in language acquisition and develop- 
ment, with a later emphasis on & more 
balanced interaction between the two. 

According-lo^PiageirCognition develop^s^ 

a result of experience. He believes That 
although language contributes to further 
development, iris the use of language that is 
determined by development and not the 
converse. 26 

Cummins ( 1979), in a summary of research n 
evidence on the role of language and cogni- 
tive development, concludes that: 
i X 
the ie\el of competence bilingual children 
achieve in their two languages acts as an inter- 
♦vening \anable in mediating the effects of their 
bilingual learning experiences on cognition Spe- 
cifically, there may bathreshold levels of kngins- 
tic compeience which bilingual children must 
attain both in order to avoid cognitive deficits 
and to allow the potentially beneficial aspects of 

/ 



becoming bilingual to influence their cognitive 

growth r 
* 

DeAvtla et al. < 1979) &ate that: 

"Edmonds ( I976)*has recently argued that a full 
understanding ol language acquisition w ill not 
* emerge until the process is viewed within a larger 
developmental f fame work And. related to 
this. 'Tremaine (1975) has examined 'syntax as 
an instance of operational intelligence* defined in 
the Piagetian sense The results indicated that 
children at the operational level performed sig- 
nificantly better in terms of 'syntax comprehen- 
sion than children classified as nonopcra- 
Uonal " N Later. DeAvila interprets Tremaines 
findings as follows "What this means is .that 
solutions which focus on English language defi- 
cits will be of limited success as long as develop- 
mental factors are not taken into account 

Studies have focused on several of these 
complex relationships. DeAvila et'al. (1979) 
examined the relationship between the degree 
of bilingualism (relative linguistic proficiency 
in English and Spanish), level of intellectual 
development (cognition), and performance 
on. two tests of cognitive-perceptual function- 
ing or field dependence independence. 

DeAvila concludes that: 

In terms of educational implications, the most 
accurate and least Value-laden interpretation of 
the findings would be to conclude that there 
seems to be a positive interaction between rela- 
tive linguistic proficiency and cognitive percept * 
tual functioning. ^ 

In summary. Tucker (1979) of the Center 
for Applied Linguistips makes the following 
comments, with which the NCES/ORA 
agrees: 1 ^ 

Nor. in mv opinion, have we managed to devise 
appropriate and valid instruments to assess lan- 
guage proficiencv What docs it mean to know 
and to be able to communicate effectively and 
acceptably in a language'' Docs there exist some 
necessary (measurable) threshold* of target lap- 

. guage proficiency which must be attained before 
one is able to profit from instruction in that lan- 
guage'* Obviously a great deal of additional 

/ interdisciplinary research is needed to examine 
the effects of factors such as intellectual poten- 
tial, social status, phvsical or emotional, devcl- 



/ opment. age~"of entry, presence of native speak- 
ers, community stereotypes, teacher character- 
istics, classroom techniques, sequencing of m 
languages, and social setting on the desirability 
♦and efficacy of bilingual education programs. I 
remain optimistic that the proposed Center for 
'Biltfigual Research may begin to move us in the 
right direction. 12 \ 

Troike ( 1980) suggests that the effect of the * 
density of a specific language minority group 
upon language proficfency in L, or L 2 is 
another factor that deserves additional re- 
search attention. . 

NCES/ORA cannot determine the effect 
of the cognitive components in the LM&AI 
on the test scores based upon the currently 
available information from discussions tyith 
experts in language development and assess- 
ment, and a review of pertinent literature. A 
post hoc study of the cognitive component 
could be completed using a sample of sub- 
jects from the population that was used for 
calibrating the LM&AI, but at additional 
cost to the Government. However, the qual- 
ity of the results of such $ study would prob- 
ably not warrant the cost since tests of lan- 
guage proficiency are generally confounded 
with language and other factors. 

Recommendation 

* NCES/ORA recommends^that NIEstate 
in the final CESS report the caveats, found in 
the discussion of this issue. There ^'re clearly 
limitations to the CESS results which are a, 
function of the current state of the'art in lan- 
guage proficiency assessment. 

Issue: Were the cut-off (critical) 
scores for the LM&AI, 
which were determined and 1 
used to classify children* as 
either English proficient or of 
limited English proficiency, 
set properly? 

discussion 

The purpose of the LM&AI was to pro- 
vide a piechanism for categorizing a child'as 
being either English* proficient or limited 
English pr6ficient. Therefore, the critical 
score determined for ^ch age-level test of the 

VJ 
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LM&AI is essential for determining valid 
LEP counts. The critical score was that 
which bfcst differentiated LEP children from 
fluent English speaking (FES) children who 
were clearly profiting .from instruction in 
? English* As an example, if the critical'score 
- on each age-^evel test is lowered by two items, 
the estimated count of LEPchildren decreases . 
from 2.41 million to 2.13 million children or 
a decreased 280,000. Similarly, if the critical, 
score for each age-level test is raised by two ! 
items, tjie estimated count of LEP children is 
increa§ed from 2.41 to 2.62 million children, 
or an increase of 210,000. Thus, a score dif- 
ference of four items has the effect of altering 
the count by nearly one-half million. 

The NCES/ORA requested the raw data 
on the student scores from Field Test 111^ 
which were used to determine the critical ' • 
scores from the LM&AI— from the prime 
contractor, L. Miranda and "Associates, Inc. 
Based upon an examination of these raw 
data and a comparison of these findings with 
Table A-4 of the NIE Draft Report on the 
CESS, a discrepancy in the data of Table A-4 
w&s called to the attention of the prime con- ' 
tractor. Miranda replied that the procedure 
used for determining the critical scores, based 
on a discriminant function analysis, w^" a , 
modification (Grand Mean-Constant = 
Cut-off) of the more conventional approach 
aad resulted in a more cohservatrte estimate 
-of the riumbpr of limited "English ptofipient 
children (see Miranda, 1980, p. 6). However, 
the data in Table A-4 did not reflect this con- 
servative approach. To remedy this situation, % 
Miranda submitted a Revised Table A-4 for 
inclusion in the final NIE report on CESS. A 
copy of the tabje is in Appendix C, page 36 
of this volume. 

In developing the LM&AI, five techniques 
were proposed as alternatives for determining 
, the critical scores. The five techniques 33 are 
summarized bfelow. 

1. For each age- level test determine the 
score on field Test II data that was one 
standard deviation below the mean 
score for the fluent English speakers 
(FES) group of that age. 



< . 

2. Similarly, use the score that was one 
standarcL deviation above the mean 
score for the limited English speaking 
ability (LESA— later revised to LEP) 
group of each age. 

3. Use the highest Field Test 111 LESA 
(LEJP)' score made by any individual on 
each ^ge-level test. 

4. Plot the score of LESA and FES sepa- 
rately and select the score equivalent to 
the point of intersection of the 'two 
distributions. \ 

5. Use discriminant function analysis 
' '(DFA) that consider^ subscores to 

determine a centroid, which can act as 
the critical point. 

After examining the accuracy of the vari- 
ous alternatives, DFA was chosen as the 
method for determining the critical scores. 

While NCES/ORA fully endorses the use 
of DFA, an issue remains regarding its use: 
the application of DFA without concern for 
' the differential "costs' 1 of misclassification. 
-DFA is a powerful tool in that it minimizes 
that proportion of 4he sample that is mis- 
classified, yowever, if the resulting cfassifica- r 
tion criteria (critical scores) consistently mis- 
classify one •subgroup, (e.g., LEP) at the h t ? 
expanse of thp other, a serious bias may 
result.' More explicitly, if there are actually 
N, LEP children and N 2 English proficient 
(fluent) children among the N*- = N, + N 2 
children of non-English language background * . 
households, then the cut-off score will lead to 
an unbiased classification procedure if and 
only if N 2 • Pr (Classified LlEP | Actually 
fluent) = N, • Pr (Classified fluent | Actually 
LEP). That is, the expected number of fluent 
children misclassified as LEP must equal the 
expected number of LEP children misclassi- 
fied as fluent. 

In defense of the procedures used , since N , 
and N 2 were not known in advance, minimiz- 
ing the overall misclassification error makes 
reasonable sense. However, as can be seen in 
Table A, the discrimination procedure actu- 
ally used was much more likely to misclassify 
LEP children than fluent children. This 
explains why the critical scores for DFA 
seemed ft>w. 34 
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_^ Table A * 

"7 

Estimated Conditional Probabilities of Coxrect and Incorrect 
Classifications by the LM&AI (See Appendix C, Revised Table A-4) 



Age 


P ' 

r !!' 




P 0 


I** 


5 


«• 0811 


0.000 


0.189 


1.000 


6 


W 0.795 " 


0.000' 


0.205 


1.000 


v 7 


0.806 


0.000 


0.194 • 


1.000 \ 


8 


- 0.893 • 


0.000 


U.1U/ 




9 


0.813 


0.000 


0.188 


1.000 - 


10 


0.833 


0.000 ' 


0.167 


1.000 w 


11 


0.682 


0.000 


0.318 


LOW 


12 


0.864 . 


. 0.182 


0.136 ( 


0.818 * 


18 


0.800 


0.000 


0.200 


1.000 


14 


0.879 


0.204 

• 


0.121 


0.796 


c. P 21 = 


Pr(Classified LEP 1 Actually LEP). 
Pr(Classified LEP \ Actually Fluent). 
PrXClassified Fluent | Actually LEP). 







d. P^ = Pr (Classified Fluent \ 



Table A presents the estimated conditional 
probabilities of correct aftd incorrect classifi- 
cations by the LM&AI for the critical scores 
fouhd irvlhe Revised Table A-4 (Appendix 

Q. 1 y 

Thfc bias evident in Table A led NCES/ 
ORA to conclude that the critical scores for 
-each age-level test of the LM&AI should be 
revised in order to remove the estimated bias, 
once estimates of and N 2 have been com- 
pijjed.'Mhc mechanism by which this can be 
done k follows: 35 

Let P,„ P r2 , P 2 „ and P 22 be defined as in 
Table A. Let N, and N 2 be the actual 
, number of LEP and fluent children, 
respectively. Finally let L and F be the 
expected number of LEP- and fluent chil- 
dren, respectively, as estimated by the 
. LM&AI. Then, 

L = N l P ll +N 2 P l2i 

F = N,P 2 , +N 2 P 22 * 

Solving for N, and N 2 , • . 

N, = (LP 22 - FP )2 )/(P M P 22 - P I2 P 2I ) and 

N 2 = (FP I1 + LP 2I )/(P 1I P 22 -P I ' 2 P 2I ) 



Of ccftirse, the values of P n , P, 2 , P 21 , and 
P 22 are functions of the actual critical scores 
that are used for distinguishing between l^EP 
and fluent children. This means that an itera- 
tive procedure must be used to determine the 
unbiased estimates of N, and N 2 based on 
critical scores associated with "balanced" 
reclassification errors. To accomplish this, 
the estimated "miscltfssification balance," de- 
fined by] L P l2 - F-P ]2 \ , must hp calculated 
for each possible critical score. For each age 
group, the critical score is selected that min- 
imizes the estimated misclassification im- 
balance. Using the expected number of LEP 
(L) and fluent (F) children and the revised 
probabilities (Ps) once the expected misclas- 
sification imbalance has been minimized, we 
can approximate the "unbiased" values of N, 
and N 2 for each age group. The values of L, 
F, P n , P 21 , and P 22 which were used to com- 

■ pute N, and N 2 are found in Appendix D. 

The results of these commutations (shown 
in Table B) clearly demonstrate the con- 
sistent bias in the ilM&AI classification 
procedure. The CESS/ LM&AI LEP counts 

-underestimate the "true" values at every age, 
except for ages 12 and 14. 
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Effect on 1978 CESS LEP Counts of Removing the Estimated Bias 





ly to l^ILrOo JLILr v^OUIll 

1 


Unbiased LEP Count 


5 


192,297 


249 734 


6 


291,622 


306,970 


7 


275,924 


320,774 


' 8 


257,807 


277,425 


•9i 


167,304 ' -v 


1 v . 189,277 ^ 


10 


294,156 


. . 329,047 


1-1 


190,064 


266,706 


12 


251,680 


207,388 . ' 


13 


196,577 


227,732 * 


14 


291,444 


246,282 


4 Total 


2;408^875 


2,621,332 



hi actual practice, the LEP counts deter- 
mined by the critical score will almost always 
differ from the "unbiased " estimate, since all 
children with a given score must fall orf one 
side of the critical score orjthe other. There- 
ford, some bias must be accepted in the 
counts, but NCES/ORA has minimized the 
expected bias by using the procedure just 
described. Table C contains the CESS Draft 
Report critical scores (with the slight modifi- 
cation mentioned earlier), the revised critical 
scores, and the resulting LEP count for each 



age level. Note that the national LEP figure 
of 2,631,075 children (Table C) compares to 
an "urfbiased" estimate of 2,621,332 (Tabic 
B). 

By minimizing the estimated bias, a less 
conservative, yet more analytically sound, 
LEP count results with a change in the 
national CESS estimate from 2,408,875 LEP 
children to 2,631,075. This change represents 
a national increase of 9.22% in the number of 
LEP children estimated in the CESS Draft 
Report. 



Table C 



Revised Critical Scores and Resulting LEP Counts 



Age 


CESS Draft Report 
Critical Score 


Revised Critical Score 

* - 


Revised LEP Count 


5 


18.5 . . 


25.5 




254,657 


6 


26.5 


29.5 




303,584 


• 7 


39.5 


44.5 


✓ 


318,470 


8 


38.5 


40.5 


280,256 


9 


43.5 


46.5 , 




188,187 


10 


* 49.5 


52.5 • 




330,979 


11 


41.5 


51.5 




271,485 


12 


46.5 


„ 44.5 




. 208,426 


13 


48.5 


52.5 




\ 229,986 


14 


52.5 


' 49.5 


*■ 


245,045 


Total 






V 

i 


2,631,075 . ' 
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Recommendation 

NCES/ORA recommends that the NIE 
final report on CESS reflect this analysis and 
the revised LEP county found in Table C. 
t 

Issue: What wm rtw i fCiTiTi^ non- 
response bias on the counts 
and estimate of the number 
of LEP children? 

Discussion 

The -question is whether non-respondents 
are similar to or different from respondents 
to the study. There is no evidence in the N1E 
Draft Report Of November 1979*to indicate 
that non-response bias was empirically in- 
vestigated* 

Donald Rogers, Vice President for Opera- 
tions with Resource Development Institute 
(one of the subcontractors), completed a 



brief paper in response to Al/CESS. lln his 
paper, Rogers presents "the results of a very, 
very simple analysis of the effects of non- 
response during the CESS study." 36 (See 
Appendix B, page 32 oHhis volume.) 

Rogers states in a letter that accompanied 
his paper that: 

My assumptions [Appendix B] generated a 
weighted LESA [LEP] total that fell within the* 

' 95% confidence interval for the total weighted 
U.S. LESA [LEP] count reported by the CESS 

* study. I do not believe that a study of non- - 
respondents will greatly increase or decrease the 
total, weighted U.S. LESA [LEP] count.* 7 

Recommendation - , 

NCES/ORA concurs v with Rogers' posi- 
tion that further investfgations of nbri- 
response bias associated \yith the 1978 CESS 
are not warranted. ■ _ 
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1. David D. Dubo& "Analytical-Issues Re- 
garding ^Children i English and Serv- 
ices Study" photoebpied material (Wash- 
ington, D.C.: U.S. Department of Educa- 
tion, National Center for Education 
Statistics, 1980), p. 1; also see this 
volume. 

2. < Ibid., pp. 2-3. 

3. T.D. Sterling and J.J. Weinkam, "What 
Happens When Major Errors are Discov- 
ered Long After an Important Report 
Has Been Published?" (paper presented 
to the American Statistical Association 
Annual Meeting in Washington, D.C. on 
August 16, 1979), photocopied material 
(Burhaby, B.C.: Simon Fraser Univer- 
sity, 1979), p. 2. 

4. Ibid., p. 12.- 

5. Rudolph C. Troike (in a personal com- 
munication with David D. Dubois, May 

> 30, 1980) suggests that a restatement of 
this issue should not detract attention 
from the fact that there is a dearth of 
basic research on the question of \#hich 
types of items are appropriate for lan- 
guage assessment and measurement at 



each age level. For example, it is neces- 
sary to examine th&*ange of grammatical 
or semantic variations that are tolerable 
for each test item at each age level. Oul$ 
after examining this question and others, 
says Troike, can wVhope to be confident 
of obtaining reliable and valid measures 
of language proficiency. ^ 

6. L. Miranda and Associates, Inc., "Re- 
sponse to the Office of Research and 
Analysis of the National Center for Edu- 
cation Statistics' Inquiry on Three Ana- 
lytical Issues Associated with the 1978 
Children's English and Services Study," 
(Bethesda, Md.: L. Miranda and Asso- 
ciates, 1980), p. 2; also see this volume. 

7. Ibid., p. 2. * 

8. J. Michael O'Malley, response to AI/ 
CESS in the form of a letter dated Janu- 
ary 3 1 , 1980 at Alexandria, Virginia, p. 2. 
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Ibid., 
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18. Dubois, 1980, p. 7. 

19. James Cummins, "Linguistic Interde- 
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, 1979>:227. 
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Linguistic, and Teacher Perception Meas- 
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hart and Winston, 1972), p. 226. 

25. Ibid. ' " 
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Appendix A 

Analytical Issues Regarding the 
Children's English and Services Study 

David D. Dubois, Policy Analyst 
x Office of kesearch and Analysis 
National Center for Education Statistics 
January 21, 1980 



Introduction 

The 1978 Children's English and Services 
Study (CE$S) was recently completed under 
contract from the National Institute of Edu- 
cation (N1E), with shared support from the 
National Center for Education Statistics 
(NCES) and the U.S. Office of Education 
(USOE). The final project report is published 
by N1E, through the National Clearinghouse 
for Bilingual Education. The principaj objec- 
tive of the CESS was- to determine an esti<j 
mate of the number of limited English profi- 
cient (LEP) children* between the ages of 5 
and 14 in the United States." 

This paper discusses three analytical issues 
that have been identified as a result of an 
assessment of the research design, data ana- 
lyses,knd other information described in the 

1978 CESS Draft Report of September 6, 

1979 (and a later revision dated November 
1979) entitled "Language Minority Children 
Tvith Limited English Proficiency in the Uni- 
ted States: Spring 1978." 

This inquiry is sponsored by the Office of 
Research and Analysis (ORA) of the Na- 
tional Center for Education Statistics (NCES). 
To date, reviewers have included the NCES 
Assistant Administrator for Research and 
✓ Analysis, the ORA Policy Analyst,- and an 
external consultant from the American Insti- 
tute for Research in Xhe Behavioral Scienpes 
(AIR) whose services were obtained under 
contract' yith the NCES/AIR Statistical 
Analysis Group in Education. This paper is 
based entirely upon these reviews. 

Recipients of this paper are invited to 
respond to the analytical issues. Based upon 
their responses and other information, the 
Office of Research and Analysis will publish 



a posiaoh-papsr *t the resection of the . 
identified" analytical issues. 

Objective of the Inquiry 

' From an analytical point of view, the 1978 
CESS could become a landmark in the 
determination of estimates of the .number of 
LEP children in the United States. The 
CESS estimate of the number of LEP chil- 
dren was accomplished directly by develop- 
ing and administering a domain-referenced 
content test to a sample of children from lan- 
guage minority households in order to assess 
language skills % in speaking, reading, writing, 
and understanding English. Before 19,78, 
estimates of this type were derived by using 
surrogate or indirect measures. f 

It is anUcipated th^t the results of the 1978 
CESS will be used extensively and frequently 
cited by U.S. Government officials, members 
of the U.S. Congress„and others. At NCES, 
for examplejt is'anticipatechthat the CESS 
data base will be used, with other surrogate* 
measures, to calibrate the 1980 U.S. Census 
data in order to determine recent and accu- 
rate LEP person counts. Additionally, the 
CESS data base will be a component of the 
NCES study to determine projections of the 
numbers of LEP persons in the United States 
for ttje next 5, 10, 15, and 20 years. 
i Sinbe the results of the 1978 QESS are of 
tremendous importance to present and future 
research studies, bilingual program and pol- 
icy development, and funding for bilingual 
education, unresolved analytical issues that 
could adversely affect the validity of the, 
results are being stated with the hope of their 
resolution. As mentioned earlier, the ORA 
will publish a position paper as a result of 
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this inquiry, to provide a technical reply to 
each issue. The reply is expected to include 
recommendations or suggestions for addi- 
tional research tasks and /or caveats to cur- 
rent CESS reports which could, in the opin- 
ion of NCES/ORA, improve the quality of 
the existing products. 

Invitation to Respond 

Recipients of this paper are encouraged to 
respond to the' issues. Respondents are 
assured 4hat their contributions will be care- 
fully considered prior to the development and 
issuance of the ORA position paper. The 



position paper will be released bnly after each 
recipient (or his or her designate ) has 
responded or has indicated that he or she will 
not ^respond to the issues. 

Written replies must be received no later 
than the close of business, Friday February 8, 
1980. Replies to the issues must be written 
and should be addressed to: 

Dr. David D. Dubois, Policy Analyst 
National Center for Education Statistics 
400 Maryland Avenue, SW, Room 3153 
Washington, DC 20202 
(Telephone: 202-245-8233) 



The persons* listed below were designated to receive a copy of this paper: 



Name 

Edward Bryant 
Lois-ellin Datta 
Karen Dietz/Pon Rogers 
. Josufe M. Gonzilez 
Ron Hall 

Ty Hartwell 
Reynaldo Macias 
Josi Martinez 
Lourdes Miranda-King 
J. Michael O'Malley 
Samuel Peng 
Leslie Silverman 
Kathy Truex 
James Vanecko ^ 

Carl Wislcr 



Agency 

Westat, Inc. 

National Institute of Education 

University of Texas- Austin/ Resource Development Institute 
Officcof Bilingual Education, U.S. Office of Education 
Office of the Assistant Secretary for Education 

(Policy Development) 
Research Triangle Institute 
National Institute of Education 
California State Department of Education 
L. Miranda and Associates, Inc. 
National Institute of Education 
Westat, Inc. 

National Center for Education Statistics 

Office of the Assistant Secretary for Planning and Evaluation 

Office of the Assistant Secretary for Education 

(Policy Development) 
Office of Evaluation and Dissemination, 

U.S. Office of Education 



History of the 1978 CESS 

The 1978 CESS was developed by NIE 
through a contract with L. Miranda and 
Associates, Inc. Lourdes Miranda-King was 
the project director, J. Michael O'Malley was 
the NIE project officer, and Leslie J. Silver- 
man was the NCES coordinator. Subcon- 
tractors included Westat, Resource. Devel- 
opment Institute, and Research Triangle 
Institute. * 

The primary mission of the 1978 CESS 



was to determine an objective estimate of the N 
number of LEP children, aged 5-14, in- 
clusive, in ttiTtflflted States. The study sur-, 
veyed p nationally representative sapiple of 
households during Spring L978, identifying 
those where a language other than English 
was spoken and where children between^he 
ages of 5 and 15 were living. T^ie Language 
Measurement and Assessment, Tnyentory 
(LM&AI),'a test in English that determines 
whether or not a child is limited in English 
language proficiency, was developed and 
\ 



administered individual^ to selected children 
from the identified households. Specifica- 
tions for the survey design and the LM&AI 
were provided b> an advisory group com- 
posed of State Education Agenc> representa- 
tives in bilingual education, assessment, and 
data collection. 

The LM&AI was designed to measure 
skills la speaking, understanding, reading, 
and writing in English. 1~he test is domain 
referenced for objectives that children aged 
5-14 would be expected to perform in order 
to profit from instruction in an all-English 
language educational environment. 

Ten separate tests, one for each age' were 
developed and used in the survey. Reliabili- 
\ ties of the test for the separate forms range 
from .86 to .92. As a result of preliminary 
field tests of the LM&AI, a critical score for 
each age-level test was determined which 
could be used to classify each child as prpfi- 
cient in English or as limited English pro- 
ficient. 

The contractor provided three caveats 
regarding the LM&AI. First, the LM&AI 
was not designed to determine placement or 
diagnosis with individual children in educa- 
tional Settings. Second, the instrument was 
designed in a manner that resulted in an 
unknown level of cultural bias. Third, the 
LM&AI items are not "pure" measures of 
English language proficiency; some of the 
items assess English language- proficiency, 
memory, and cognitive ability. 

Statement and Discussion of the Issues 

Three analytical issues are presented and, 
discussed: 

1. Were the items which were selected for 
inclusion in the Language Measurement 
and Assessment Inventory (LM&AI) 
selected properly? 

2. We re the cut-off scores for the LM&AI, 
which were determined and used to 
classify children as either English profi- 
cient or of Kmited English proficiency, 
set properly? 

3. What were Ihe effects of non-response 
bias on the counts and estimates of the 
number of LEP children? 



If the first question is answered negatively, 
then the value of the entire 1978 CESS is 
broughunto question. In the event that it is 
answered affirmatively, then a 'negative an- 
swer to the second question would imply the 
need for further analysis of the CESS data— 
and possibly for the collection of additional 
data — in order to recompute the cut-off 
scores. The issue raised by the third question , 
could be empirically investigated in the event 
that it was decided to collect the additional 
data described earlier. 

A detailed discussion of each issue follows. 

Issue: Were the items which were 
** selected for inclusion in the 
Language Measurement and 
Assessment Inventory (LM& 
AI) selected properly? 

Discussion 

>Each age-level instrument of the LM&AI 
consisted of a set of items that could be 
scored so that a high score would indicate 
that the child was proficient in English while 
a low score would indicate that the child was 
* limited English proficient. Therefore, the 
issue can be rephrased in the following 
manner. Is English language proficiency the 
dimension on which the scores vary, or are 
other dimensions associated with variation in 
the scores? More specifically: 

• Are the test scores related to language 
dominance? 

• Are the tgst scores related to genteral 
language development? 

The question of language dominance is 
addressed in the project Draft Report (No- 
vember 1979): 

English should be the exclusive criterion irre- 
spective of the child's proficiency m the non- 
English language. Thus, language dominance 
was considered irrelevant to the discussion, 
(page 11-3) 

The objective of the study is subject to 
question on the basis that, for bilingual edu- 
cation policy development, a child's domi- 
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- nam language nrfght affect the potential 
benefits he or she could derive from partici- 
• pating in a bilingual education program. The 
reader is cautioned that this review intention- 
ally docs nolattempt to define operat^nally 
the phrase "bilingual education program.** 
Whether it is appropriate to assess English 
language proficiency while ignoring the child's 
proficiency in another language remains a 
policy question to be addressed. 

Are the scores on the test related to general 
language development? The project Draft 
Report (November 1979) states that: 

items on the test arc not "pure" measures of 
English language proficiency. In some cases, the 
items assess English language proficiency, mem- 
ory, and cognitive ability. The intermingling of 
the potentially disparate constructs was inten- 
tional to give the items as much validity for 
representing important school tasks as possible 
(page A- 10) 

Any test so developed could also differen- 
tiate between two children with equal English 
language proficiency, giving a higher score to 
the child with greater memory or cognitive 
abilities. It could be argued, therefore, that 
the^test development procedures should have 
excluded items not primarily associated with 
4 English language proficiency. The types of 

items selected for the test (Draft Report, 
November 1979, Table A-l) appear generally 
to assess relevant content. There is, however, 
% a component of general cognitive jdevelop- 
ment, not merely English language devel- 
opment. 

The choice of items for the LM&AI was a 
function of a field test. Items were selected 
that best differentiated between two criterion 
groups. The project Draft Report (November 
1979) states: 

The test was. 'being developed to differentiate 
language minorities who were limited in English 
proficiency from those who could profit from 
instruction in English Items under development 
were to be field tested with two clearly defined 
criterion groups, (aj limited "English proficient 
children, and (b) fluent English speaking child- 
ren who were clearly profitigg from instruction 
in English, (page li-6) 
• 
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The test was clearly being prepared for 
administration to language minority chil- 
dren. The dimension being tested was essen- 
tially that on which those two groups differed 
most. It could be argued that the two groups 
differed on native language as well as English 
language proficiency and, therefore, the test 
scores could be expected to have a partial 
language dominance loading. A solution to 
this problem might be to equate the two cri- 
terion groups on proficiency in a non-English 
language, making the test independent of 
language dominance. 

Issue: Were the cut-off scores for 
the LM&AI, which were deter- 
mined and used to classify 
children as either English 
proficient or of limited Eng- 
lish proficiency, set properly? 

Discussion 

The purpose of the LM&AI was to pro- 
vide a mechanism for determining whether a 
child was either English proficient or limited 
English proficient. Therefore, the cut-off 
score chosen for each age-level test of the 
LM&AI is criticaPRJr determining valid 
counts. As an example, if the cut-off score on 
each age-level test is lowered by two items, 
the estimated count 'of LEP children drops 
from 2.41 million children to 2.13 millfcn, a 
decrease of 280,000. Similarly, if the cut-off 
score for each age-level test is raised by two 
items, the estimated count of LEP children 
rises from 2.41 million children to 2.62 mil- 
lion, or an increase of 210,000. Thus, a score 
difference of four items has the effect qf alter- 
ing the count by nearly one-half million. 

Recall that the cut-off score was that score 
which best differentiated LEP children from 
fluent English speaking (FES) children who 
were clearly profiting from instruction in 
English. 

In developing the LM&AI, five techniques 
were proposed as alternatives for determin- 
ing the cutoff scores. The five techniques are 
summarized on page 38 of the Children's 
English and Services Study: Technical Report 
on the LM&AI (L. Miranda and Associates, 
Inc., September 10, 1979): 
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1. For each age-level test determine the 
score on Field Test 11 data which was 
one standard deviation beloAv the mean 
score for the FES group of that age. 

2. Similarly, use that score which was one 
standard deviation above the mean 
score for the LESA (later revised to 
LEP) group of each age. - 

3. Use the highest Field Test III LESA 
score made by any individual on each 
age group test. 

4. Plot the scores of LESA and FES 
separately and select the score equival- 
ent to the point of intersection of the 
two distributions. 

5. Use discriminant function analysis 
(DFA), which considers subscores to 
determine a centroid point that can act 
as the critical point. 

After examining the accuracy of the vari- 
ous alternatives, DFA was chosen as the 
method for determining the cut-off scores. 
Whtle this Office fully endorses this choice, 
three subissues remain of concern.* 

First, the preceding excerpt from the 
Technical Report implies that the subscojEs 
were used in the DFA. If this is so, several 
events must have happened. 

1. The subscores would be transformed 
into a new total score representing a lin- 
ear combination of the subscores. This 
new score would be roal valued (as 
opposed to integer valued) and it would 
be conceivable— in fact, highly likely — 
that relative scores between two indi- 
viduals could be reversed. That is^ if 
individual A had a higher original score 
than individual B, the revised DFA 
score for A could easily be lower than 
that of B due to differential weighting of 



the subscores. Since no scores on the 
final CESS tape are non-integer valued, 
and since no reversal of the kind »dis- 
cussed above occurred, one can only 
assume thpt subscores were, in fact, not 
^ used in th? DFA, 

2 The relative weighting of the items,v 
which was carefully designed, would be 
totally revised by the differential weight- 
ing of the DFA procedure. This is 
another reason that this Office doubts 
that subscores were used. 

The second subissue concerns the applied 
tion of , DFA without concern for the differ- 
ential "c^st" of misclassification. DFA is a 
powerful tool in that it minimizes the total 
proportion x of the sample that is misclassified. 
However, if the resulting classification criteria 
(cut-off scbre$) consistently misclassify one 
subgroup (e.g.\ LEP) at the expense of the 
other, a seriou^bias may result. More ex- 
plicitly, if there ate actually N, LEP children 
and N 2 English proficient (fluent) children 
among the N = N, + N 2 children of non- 
English, language background households, 
then the cut-off score will lead to an unbiased 
classification procedure if and only if N 2 • 
Pr (Classified LEP \\ Actually fluent) = N, • 
Pr (Classified fluent |j Actually LEP). That is, 
the expected numbef of fluent children mis- 
classified as LEP must equal the expected 
number of LEP children misclassified as 
fluent. J 

In defense of the procedure used, since N, 
and N 2 were not known a priori, minimizing 
the overall misclassification error makes 
reasonable sense, powever, as can be seen in 
Table A', the actual discrimination procedure 
used was much more .likely to misclassify 
LEP children than fluent children. This 
explains why the cut-off scores for DFA 
seemed low (see page 43 of the aforemen- 
tioned Technical Report). 



♦The analyses found here were developed by Dr. Rolf M. Wulfsbcrg, the Assistant Administrator for 
Research and Analysis at NCES. 
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Table A 

Estimated Conditional Probabilities of Correct 
and Incorrect Classifications by the LM&AI 



— . — « \ 






P * 


P b 


P c 


P d 


5 ' 


0.892 


0.000 


0.108 


1.000 


6 


0.955 . 


0.037 


0.045 


: 0.963 


7 


0.889 


0.000 


0.111 


1.000 


S 


0.929 


0.000 


0.071 


'•• ' 1.000 




0.906 • 


• 0.000 




(V\r\ 


10 


0.944" 


0.000 


0.056 . 


. . 1.000 


11 


0.795 


0.000 


0.205 


1.000 


12 


0.864 


0.182 


0.136 


\ 0.818 


13 


0.880 ' 


, 0.000 


0.120 


• 1.000 


14 


0.879 . 


0.204 ' 


0.121 


0.796 


a. P„ 


= Pr (Classified LEP | 


Actually LEP). 






b. P„ 


= Pr, (Classified LEP j Actually Fluent). . 






c. P 21 


= Pr (Classified Fluent 


1 Actually LEP). 






d. P 22 


= Pr (Classified Fluent 


| Actually Fluent). 







ased values of N, and N 2 f° r eac h a 8 e group. 
The results, which are shown in Table B, 
clearly demonstrate the cdhsistent bias in the 
LM&AI classification procedure % The CESS/ 
LM&AI LEP counts underestimate the 
"true" values at every age, except for age 14. 

If we accept the new LEP count as more 
realistic estimates of the true values, then we 
can adjust the cut-off scores to reflect these 
new counts by raising (except for age 14) the 
cut-off scores until the proper number of 
children have been classified as LEP. In re- 
alky, this point will usually fall in the middle 
of a cell (score), so one can choose the cut-off 
score that will yield the closest estimate to N,. 

In the case presented in T^ble C, a differ- 
erit rule was used. Since the data show an 
abnormal "roller-coaster*' effect in the rela- 
tionship between age and percent LEP, the 
cut-off score leading to the percentage closest 
to the overall mean percentage was chosen 
for each age group. That is, the lower cut-off 
score was generally uj^ed for even ages and the 
higher cut-off score was generally used for 
odd ages. 



The evident bias described above raises the 
third subissue: Should the cut-off scores be 
revised to remove the estimated bias after the 
fact (when we have estimates of N, and N 2 )? 
This Office tends to feel that this should be 
done, perhaps by the mechanism described 
below. 

Let P,,, P l2 , P 2I , ancj P 22 be defined as in 
Table A. LeLN, and N 2 ^ e ^ e actual 
number of LEP and fluent children, 
respectively. Finally let L and F be the 
expected number of LEP and fluent qjiil- 
■ dren, respectively, as estimated by the 
LM&AI. Then, 

L = N,P M +N 2 P I2 
F = N I P 2I + N 2 P 22 
Solving for N, and N 2 , 
N l MLP 22 -FP l ' 2 )/(P II P 22 -^P 2l )and 
N 2 = (FP IJ + LP 2I )/(P JI P 22 -P J2 P 2I ) 

By using the actual CESS estimates for L 
and F, we can then approximate the unbi- 
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Table B 



Effect on LEP Counts of Removing Estimated Bias 



Age 


CESS LEP Count 


Unbiased LEP Count 


5 


* 

192,297 


i 

21^580 


6 


291,622 


3017767 


7 


275,924 


310,375 


8 


257,807 


277,510 


9 


167,304 


184,662 


10 


v 294,156 


• 311,606 . 


11 


190,064 


239,074 


12 


251,680 


262,412 




n) ,96 - 577 


223,383 - 


8 


291,444 


284,766 


Total i 


f 1 2,408,875 


2,611,135 






Table C 



Modified Cut-off Scores and Resulting LEP Counts 



Age 


Old Cut-off Scores 


New Cut-off Scores 


New 
LEP Count 


5 


18.5 - 


21.5 


223,327 


6 


26.5 


28.5 


298,929 


7 • 


39>5 


43.5 


307,759 


8 * 


38.5 


-39.5 


268,830 


9 


43.5 


46.5 


188,187 


10 


49.5 


50.5 


31Q;860 


11 


, 41.5 


45.5 


246,921 


12 


% N 46.5 


J 46.5 


251,680 


13 


, 48.5 


51.5 


223,785 


14 


52.5 ' 


52.5 


291,444 






_ I ' 




Total 




2,611,722 



The relationships among the CESS/ 
LM&AI estimates, the unbiased estimates, 
and the adjusted CESS estimates are evident 
in Chart 1, following. Chart 1 shows the per- 
centage of each age cohort for each of the 
estimates of the number of limited English 
proficient children. 



Applying the procedure described above 
should go a long way toward removing the 
bias in the LM&AI. Of course, the values of 
P M used in the derivation are conditioned on 
the original cut-off scores used by the 
LM&AI. With the modified cut-off scores, 
the P, ,s would change (as do the new L and F 
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counts shown in Table C), so that the results 
could still change slightly. (This is because 
the LM&AI sample of 354 fluent children 
and 337 LEP children are not necessarily 
representative of their respective popula- 
tions.) The Office of Research and Analysis is 
attempting to obtain the original data used to 
determine the cut-off scores on the LM&AI 
from L. Miranda and Associates, Inc. jn 
order to explore this issue. 

Issue: What were the effects of non- 
response bias on the counts 
and estimare of the number 
of LEP children? 

Discussion * ^ 

In survey research of this type, the poten- 
tial effects of non-response bias are a reality. 
The question to be addressed is whether non- 
respfondents arc similar to or different from 
respondents to the study. 

Response rates by regional subpopulations 
(New York, Texas, California, and remainder 
of the United States) for the household 
screener, household questionnaire, and ad- 
ministration of the LM&AI are presented in 
Table III— 1 of the Draft Report (November 




1979). From the table it can be determined 
that the response rates, totaled over all sub- 
populations, were: household screener, 76.2%; 
household questionnaire, 93.8%; LM&AI 
administration, 84.6%. Response rates were 
derived by using ,the formula 

Response Rate = Total Number Completed x m 
Total Number Eligible 

There is no evidence in the Draft Report 
(November 1979) to indicate that 'non- 
response bias was empirically investigated. 
Although adjusting weights by poststralifica- 
tion is customary, this is not necessarily a 
satisfactory substitute for empirically investi- 
gating differences between respondents and 
non-respondents. 

If it is determined that the items included 
in the LM&AI were indeed selected properly 
and, additionally, a decision is made to col- 
lect additional data for recalibrating the 
LM&AI, an empirical investigation of non- 
response bias can be undertaken concur- 
rently. 

In summary, ORA reviewers betiev* that 
these issues can be resolved and, accordingly, 
that the study can be retained by cooperative, 
responsible action. 
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Append^ B 



Non-response Analysis 



' Donald Rogers 
Resource Eyelopment Institute (Austin, Texas) 



February 1980 



Purpose 

The purpose of this paper is to present the 
results of a very; very simple analysis of the 
effects of non-response during the CESS. 

General Procedure 

The general procedure was to assume that 
non -responding *SRC iQfomplete; probable 
ineligible households (Code 8 households)" 
had characteristics that were significantly dif- 
ferent from responding households. The 
impact of this assumption was then deter- 
mined by reweighting the data and recomput- 
ing Non-English Language Background 
(NELB) and' Limited English Speaking Abil- 
ity (LESA) counts. 



Limitations 

The analysis reported here uses average 
weights. Ideally, each stratum is considered 
individually. However, the resources required 
for a sfratum-by-stratum analysis were not 
available. Therefore average weights were 
used because they were easy to compute. This 
means that the results of this analysis only 
indicate or suggest the type of results that 
would' be obtained by a sophisticated 
analysis. 



References 

This paper is based on the information 
presented in Resource Development Insti- 
tute's (RDI) reports. Data have been taken 
from Sectidn 8 (Data Analysis Procedures) 



and Section 9 (Results) of Volume 1. Weight- 
ing formulae are taken from Appendix 6.6 of 
Volume II. The reader must have these 
reports to follow this paper. For example, the 
definitions- of variables are presented in 
Appendix 6.6 and are not repeated here. 



Assumptions 

The. following assumptions were made to 
assess the ^ffects of non-response: 

1. All Category 8 households complete the 
SCR. 

2. The percentage of Category 8 house- 
holds that are eligible and complete the 
HHQ is twice as great as the percentage 
of Category 1, 2, and 3 households, 

3. All of the eligible Category 8 house- 
holds complete the HHQ. 

4. The average number sampled per eligi- 
ble household is the same for Category 

8 households. s 
* 

5. The average number of completed 
LM&AI per household is the same for 
Category 8 households. 

£. The average number of LESA children 
«per household is the same for Category 
8 households. 



The effects of these assumptions on the 
"raw" data are presented in the followfhg 
tables. 
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Tables A, B, and C 



Effects of Assumptions on Raw Data in Assessing 
* 'the Effects of Non-response 



Table A 



Household 
Codes 



Number 
Complete SCR 



Percent 
Complete HHQ 



Number 
Complete HHQ 



, I.2.&3 
8 



Totals 



25,358 
5,790' 

31,148 



6.5 
13.0 



1,652 
753 



2,405 



Table B 







Average 




♦ 




< 




Number 




Average Number 






Number 


Sampled 




of Completed 


Number 


Household 


Complete 


Per 


Number 


LM&AI 


of Completed 


Codes 


HHQ 


Household 


Sampled 


Per Household 


LM&AI 



1 r rf 

. 1.2.&3 

. 8 


1,652 1.78 
753 1.78 


i 2,953 
.1,340 


1.16 
1.16 


1,909 
873" 


Totals * 


2,405- 


1 

4,293 " . 




2,782 






> 


Table C 






Household 
Codes 


Number of 
Completed LM&AI 


Percent LESA 

- 




Number LESA 


1.2.&3 . 

8 ' 


1,909 
873 


71.24 
71.24 




1,360 
622 


\ 

Totals 


2,782 • 






' - 1,982 
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Changes irf Values 

The raw data were used to compute aver- 
age values for the variables in Appendix 6.6 
of Volume II in RDI's final CESS / reports. 
The computed values are presented^ in the 
table following; the formulae have been omit- 



ted because, they anpear in Appendix 6.6. 
Although the usef^f the symbols is not 
entirely appropriate and is not precisely con- 
sistent with the definitions presented in 
Appendix 6.6, the results are presented in this 
manner to make it easy for the reader to fol- 
low the calculations. * 



• * Table D 



Changes in Average Values Computed for Variables 



H ; ' r r ■ ■ 7 

Average or Estimated Value Average or Estimated Value 

Variable Before Assumptions After Assumptions 

. . . 1 , 



* nij 


33,283 




33,283' 


rfcf 


25,358 




\ 31,148 


ws, 


64.5 




} 64.5 


Lhs 


2,146,753 




2,146,753 




1,635,591 




2,009,046 




'• * 83.9 




68.9 


Shy f 


1,762 




2,515 


Shi) \ 


1,652 




2,405 


M hs \ 


147,832 




173,284 




\ 138,603 




165,704 




\ 89.5 




72.1 




3,084,452 - 




3,048,452 


P' 
hijm 


1 2,953 




4,293 


Chi|m 


34,061 ** 




49,573 


Qhijm 


1,032 




833 


P" 
hijm 


1,909 




2,782 


K 


3,047,496 




3.571,776 


ft;, 


1.970.088 




2.314,624 




1.597 




1,283 


Q (adjusted for SIE) 


1,997 




1 ,370 


Total NELB 


3.81 1,850 




3,811,850 



The assumption about the Category 8 
households increased the samplewiumber of 
LESAs from 1,360 to 1,982. This is approxi- 
mately a 46% increase. The effects on the 
total U.S. estimate depend upon assump- 
tions about how these cases are weighted. 
Table E reports the average weights that have 
been used to this point in the analysis. 



Analysis 

The assumptions about the Category 8 
households increased the sampled number ^of 
v NELB persons from 1,909 to 2,782. This is 
approximately a 46% increase. However, 
because of the weighting procedures, this 
increase has /^meaningful effect on the total 
U.S. estimates. 
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Table E 



Effects of Assumptions on Average Weights and U.S. Estimate 
& ■ - 

Average Weight ' U.S. Estimate 



Assumption 


Type 


Number 


Before 


NELB 


1,909 


After 


NELB 


2,782 


Before 


LESA 


1,360 


After 


LESA 


' 1,982 



1,997 
1,370 
1,771 
, Unknown 



3,811,850 
•3,811,850 
2,408,^08 
Unknown 



Table E indicates that before the assump- 
tions, the average LESA weight is less than 
the average NELB weight. The assumptions 
that have been made should not affect this 
relationship, and the average LESA weight 
should continue to be less than the average 
NELB weight. However, to test response 
bias, assume that the NELB and LESA aver- 
age weights xare the same after the assump- 
tions' and are equal to 1,370. This yields a 
total U.S. estimate* of 2,715,340 LESAs. 
This estimate is 306,432 LESAs greater than 
the LESA estimate reported by the CESS. 



However, an estimate of 2,715,340 LESAs is 
within the 95% confidence interval pf the 
total U.S. LESA estimate reported By the 
CESS*. 

Conclusion \ 

The analysis that has been reported here is 
rather simple and superficial. Some of the 
assumptions that have bfeen made border 6n 
being outrageous. Nevertheless, the results of 
the analysis indicate that the& assumptions 
do not create meaningful differences in th?^ 
final estimates. 





Appendix C 



Revised 

_J Table A-4 

Classification Errors in Selection of a Criterion Score 
on the Language Measurement and Assessment Inventory 



A op 
Age 


Proficiency in ' 
English on the 

PtaH i n t nr 
x rCUIlMJi 


. Proficiency in 
English on the 
Criterion* 
Fluent Limited 


Critical 
Score 


Percent 
Accuracy 


5 


Fluent 

T imitpH 


32 
7 


0 

* 30 


19 


90.0 


6 


Fluent 


27 
9 


0 
35 


• 26 


87.3 


7 


"Fluent 


31 • 
7 


0 
35 


39 


89.6 


8 


Fluent 
Limited 


36 
3 


0 

25 


39 


95,3 


9 


FlUent 
Limited 


35 
6 


0 
26 


43 


91.0 


10 


Fluent 
Limited 


35 
6 


0 , 
30 


49 


91.'5 


11 


Fluent 
Limited 


34 
4 14 


0 

30 


41 


82. i 


12 


Fluent 
Limited 


27 
3 


6* 
19 


,47 


83.6 


13 


Fluent 
Limited 


42 
5 


• 0 
20 ' 


48 ' 


92.5 


14 


Fluent 
Limited 


39 
4 


10 

29 " 


52 


82.9 



1 



a. Entries are* number of cases in field test three. 

b. For example, percent correct at age 5 equals 100 (32 + 30)/69 = 90.0. 



.1 0 

'36 



Appendix D 



Values of L, F, P u , P 12 , P 21 , P^, N„ and N 2 for the 
Minimized Misclassifications of LEP Children by Age Cohort 



Age 


















Cohort 


L 


F 


P„ 








A 


N 2 


5 


254,657 


73,212 


1.000 


0.063 


0.000 


0.937 


249,734 


' 78,136 


6 


303,584 


90,989 


0.963 


0.091 


0.037 


0.909 


306,970 


87,603 


7 


318,470 


144,466 


0.968 


0.056 


0.032 


0.944 


302,774 


142,162 


8 


280,2*56 


37,083 


1.000 


0.071 


0.000 


0.929 


277,422 


39,917 


9 


J 88, 187 


143,003 


0.971 


0.031 


0.029 


0.969 


189,277 


141,913 


10 


330,979 


32,565 


1.000 


0.056 


0.000 


0.944 


329,047 


34,497 


11 


271,485 


132,731 


0.971 


0.091 


0.029. 


0.909 


266,706 


137,510 


12 


208,426 


191,107 


0.879 


0.136 


0.121 


0:864 


207,388 


192,145 




-22-9,986 


94,240 


0.976 


0.080 


0.024 


0.920 


227,732 


96,494 


245,045 


241,365 


•0.877 


0.121 


0.123 


0.879 


246,282 


240,128 




Appendix E 



Contributors 



This report was completed while the 
author was assigned to the National Center 
for Education Statistics (NCES) as an Edu- 
cation Policy, Felloe sponsored by the Insti- 
tute for Educational Leadership, The George 
Washington University, and Dr. Rolf M. 
Wulfsberg, the Assistant Administrator for 
Research and Analysis at NCES. The author 
gratefully acknowledges Dr. Wiilfsberg's 
encouragement and support during tjie 
development and completion of this project. 

Dr. Roslyn A. Korb (Technical Planning 
Officer, NCES) and Dr. Donald McLaughlin 
(Statistical Analysis Group in Education, 
NCES, and the American Institutes for 
Research) provided technical opinions on the 
Childrens English and Services Study 
(CESS) in the formative stage of this method- 
ological review. 



Dr. Edward A. DeAvila (Linguametrics 
Qroup) and Dr. Rudolph G Troike (then 
Director, National Clearinghouse for Bilin- 
gual Education) provided psychometric and 
linguistic perceptions on language develop- 
ment, measurement, and assessment. 

Dr. Rebecca Oxford (InterAmerica Re- 
search Associates, Inc.) shared several refer- 
ences on Bilingual education with the author. 

Those individuals who responded to the 
issues paper (Appendix A) provided histori- 
cal information on the development of the 
CESS which was essential for obtaining an 
in-depth understanding of the study. 

Finally, Mr. Richard Haber (Division of 
Multilevel Education Statistics, NCES) com- 
pleted several programming tasks for this 
project. 
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2. Reanalysis of the Number of Limited 
English Proficient Students Estimated in the 
Children's English and Services Study 



by 

-Lawrence M. Rudner, Rosedith Sitgreaves Bowker, 
l and John W. Chambers 

National Institute of Education 
March 1981 



Establishing LM 

The National Center for Educational Sta- 
tistics (NCES), Office for Research Analysis 
(ORA) released, on August 27, 1980, a 
methodological review of the 1978 Childrer\s 
English and Services Study (CESS) con^ 
ducted by NCES and the National Institute 
of Education (N1E) in conjunction with L 
Miranda and Associates df Bethesda, Mary- 
land. The primary objective of the CESS was 
to derive an estimate of the number of chil- 
dren of limited English proficiency, aged 5- 
14, from language minority households in the 
United States. In their report, NCES devel- 
oped and applied an alternative procedure 
that resulted in a 9.229c higher estimate. This 
paper briefly examines the CESS and 
NCES ORA methods and the differences 
associated with the final estimates. 

A measure of English language profi- 
ciency, the Language Measurement and 
Assessment Inventory (LM&A1), was devel- 
oped and administered to a probability sam- 
ple of 1,900 students from language minority 
households, and total scores for each student 
were computed. Students within each age 
level scoring above a predetermined critical 
value for the age level were classified as fluent 
(F), students scoring below the value were 
classified as limited English proficient (LEP) 
' The proportion of sampled students at each 
level classified LEP was projected to the 
estimated population of language minority 
• students to estimate the prevalence of limited 
English proficiency in the population. ' 

In order to set the critical value for each 
age level, the LM&A1 was administered to a n 
validation sample of approximately 60 stu- 
dents per age group whose English abilities 
were determined by school personnel. The 
students within each group were selected so 
that approximately half of the students were 
fluent and half of them were LEP. The CESS 
then investigated five methods for determin- 
ing, from this validation sample, the critical 
values to be applied to the larger probability, 
sample. NCES ORA developed a sixth alter- 
native. 

\Different methods for determining the crit- 
ical values were examined to determine 



I Critical Values 

which would minimize the overall error of 
misclassification of students. Because of the 
overlap of high-scoring LEP and low-scoring 
fluent students, two types of error had to be 
considered: the misclassification of fluent 
students as LEP (Type 1 error), and misclas- 
sification of LEP students as fluent (Type 11 
error). A cut-off score that was too high 
would result in excessive Type 1 error; one 
that was too low would result in excessive 
Type 11 error. 

Discriminant function analysis (DFA), the 
technique used in the CESS report, is 
designed to maximize the difference in the 
discriminant function means for the two 
groups and defines the midpoint of the 
weighted mean scores of LEP and fluent stu- 
dents as the critical value. The other tech- 
niques that were considered, but rejected, 
maximized the probability of a correct clas- 
sification of LEP students at the expense of a 
large Type 1 error, or they maximized the 
probability of a correct classification of fluent 
students at the expense of a large Type II 
error, or they were inapplicable because of 
sampling limitations. The CESS report's use 
of discriminant function analysis sought to 
meet the objectives of minimizing the overall 
error of misclassification. 

NCES ORA correctly pointed out that 
the procedure resulted in a greater Type 11 
than Type 1 error. Due to heterogeneity in 
the variance of LM&A1 scores obtained by 
the sampled LEP and fluent students, the 
critical values determined by discriminant 
analysis appeared to be low An alternative 
procedure was developed that sought to 
equalize the two types of error. This equaliza- 
tion would balance the effect of misclassifica- 
tion and result in a more useful overall count 
of LEP and fluent students, even though the 
magnitude of one of the two types of error 
might be greater. NCES ORA empirically 
examined every possible critical value at each 
age level to determine which value best met 
their objectives. 

The NCES OR A and CESS critical values 
differ substantially and result in different 
population counts at each grade level Had 



the total score variances of the identified 
LEP and fluent students been equal, as is 
assumed in discriminant function analysis, 
the NCES/ORA and the discriminant analy- 
sis would have provided identical critical 
values and population counts. While the new 
t NCES/ORA method overcomes one prob- 
lem-associated with the validation sample— 
non-equal total score variances — it docs not 
result in a statistically significant difference in 
the overall estimate of LEP children. 

Accuracy of estimatibn increases as a func* 
tion of sample size. The larger the sample 
size, the more accurate the estimate. Using 
1,900 students in the probability sample pro- 
vided accurate estimates of the proportion of 
students in each age level scoring at or below 
each total score value. The sample used to 



determine critical values, however, was con- 
siderably smaller. Table I shows the amount 
of variation that would be expected as a 
result of using different samples of approxi- 
mately 60 students. The 95% confidence 
intervals, derived from the standard error of 
estimate, provide an upper and lower^boun- 
dary to the critical value estimates. Scores H 
within this band are not considered to be sig- 
nificantly different from the original CESS 
values at the .05 probability level. Due to the 
small samples, these confidence intervals are 
extremely wide. In all but four age levels, the 
NCES/ORA reVised critical values fall within 
the confidence interval. The overall NCES/ 
ORA estimate of 2.6 million LEP students 
falls within the confidence interval placed 
around the CESS estimate of 2.4 million. 



Table I 



Language Measurement and Assessment Inventory 
Validation Summary Statistics 











CESS 


95% " 


NCES/ORA 










Critical 


Confidence 


Critical 


Age 


N« 


SD'-s 


SEE 2 


Value 


Interval 3 


Value. 3, 4, 6 - 


5 


69 


12.8 


1.54 


18.5 


15.5-21.5 


25.5* 


6 


71 


16.1 


1.91 


26.5 


22.5-30.5 ' 


29.5 


7 


67 


18.0 


2.20 


39.5 


35^3.5 


44.5* 


8 


* 


15.4 


1.92 


38.5 


34.5^2.5 


40.5 


9 




20.0 


2.44 


43.5 


38.5-58.5 


46.5 


10 


71 


20.0 


2.37 


49.5 


44.5-54.5 


52.5 


11 


78 


14.6 


1.65 


41.5 ' 


38.5-44.5 


51.5* 


12 


55 


17.3 


2.33 


46.5 


41.5-51.5 


44.5 


13 


67 


13.9 


1.70. 


48.5 


45.5-51.5 


52.5* 


14 


82 


24.0 


2.65 


52.5 


47.5-57.5 


49.5 



1. SD - overall standard deviation for the combined groups. ' 

2. SEE - standard error of the critical value estimate = sd / 1 R 

3. 95% confidence interval = CESS critical value ± 1 .96 SEE since only whole number scores are 
possible. Critical values have been rounded to the nearest midpoint. 

4. Starred values fall outside the confidence interval. 

5. From Miranda (1979) 

6. From Dubois (1980b) 



The CESS report estimated the number of 
LEP children to be 63% ± 8% of all children, 
aged 5-14, living in language minority house- 
holds. 

In summary NCES/ORA achieved, in an 
alternative, empirical manner, the CESS 
investigation goal of establishing "LM&AI 
critical values that result in useful estimations 
of the number of LEP children at each age 
level. The resultant critical values on the 
LM&AI and, hence, the estimates of LEP 
students at each level, differ from the origi- 
nal CESS values because of the heterogeneity 
of variance in the validation sample. How- 



ever, in a majority of cases the NCES/ORA 
revised critical values and revised total count 
fail within the confidence intervals placed 
around the CESS values. Since the approach 
developed and applied by NCES/ORA is 
mpre in line with objectives of the CESS 
investigation and overcomes one limitation 
resulting from the relatively small sample 
size, its resultant estimates of the number of 
LEP children represent a theoretical im- 
provement o\fcr the original values. Further 
research using larger samples to obtain esti- 
mates with much smaller confidence intervals 
is highly recommended. 
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Appendix A 



Description of the Language Measurement and Assessment Inventory 

(LM&AI) ' 



The LM&AI was designed for the Chil- 
dren's English and Services Study (CESS) to 
estimate the number of limited English profi- 
cient children in the United States. The study 
provides estimates of the number of limited 
English proficient (LEP) children aged 5-14 
living in households where a language other 
than English was spoken it/ Spring 1978. 
Specifications for the survey design and for 
the LM&AI were provided by an advisory 
group composed of State Education Agency 
representatives in bilingual education, assess- 
ment, and data collection. Major results of 
the study were reported in May 1979 at the 
annual meeting of the National Association , 
for Bilingual Education. 

The LM&AI provides a single score that 
reflects overall English language proficiency 
for skills that, in the judgment of the advisory 
group, are typicall} taught in the schools and* 
are important for being able to profit from 
instruction in English. The skills include 
speaking, understanding, reading, and writ- 
ing in English. The test is individually admin- 
istered and requires 20-30 minutes to com- 
plete. LanguSge groups included during field 
tests were Arabic, Chinese, Greek, Italian, 
Japanese, Polish, Spanish (Cuban, Mexican 
American, and Puerto Rican), and Viet- 
namese as well as native English speakers. 
Items retained in the final version of the test 



met the content specifications provided by 
the advisory group, and discriminated statis- 
tically between children designated by schools 
as either limited or fluent in English language 
proficiency. There are 10 separate forms of 
the test, one for each age from 5 to 14 years. 

Reliabilities of the test for the separate 
forms range from .86 to .92. 

A criterion score for each age form was 
^selected that identified LEP children. For this 
purpose, a statistical technique was chosen 
that maximized accurate classifications based 
on the total test scores when attempting to 
predict school classifications of fluent or 
limited English proficient groups. The criter- 
ion score was used in a national CESS sam- 
ple to identify LEP children. 

The LM&AI is not intended for individual 
diagnoses or for use beyond the purpose for 
which it was designed in the CESS survey, 
nor should the LM&AI be used as a diagnos- 
tic instrument for any of the sub-skill areas in 
oral or reading proficiencies without further 
development work. 

The National Institute of Education has 
awarded copyright to the test developer, who 
should be contacted for further information 
about the instrument: L. Miranda & Associ- 
ates,- 4340 East-West Highway, Suite 906, 
Bethesda, Maryland 20014, (301) 656-8684. 
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Appendix B 



Response to 
* The Office of Research and Analysis of the 
National Center for {Education Statistics' 
Inquiry on Three Analytical Issues 
Associated with the* 1978 
Children's EnglisH and Services Study 



Submitted to: * 

T^e National Center for Education Statistics 
Office of the Assistant Secretary for Education 
Washington, D.C 20202 



Submitted by: 

L. Miranda and Associates, Inc. 
4340 East-West Highway, Suite 906 
Bethesda, Maryland 20014 
(301) 656-8684 



February 15, 1980 



February 15, 1980 

David D. Dubois 
Rolf M. Wulfsberg 

National Center for Education Statistics 
400 Maryland Avenue, S.W., Room 3153 
Washington, D.C 20202 

Dear Messrs. Dubois and Wulfsberg: 

We are pleased to present herein our response to your "Analytical Issues Regarding the 
Children's English and Services Study" paper. In order to draw on as many perspectives as 
possible, several former consultants to L. Miranda & Associates, Inc. (LM&A) during the 
CESS . project were queried on their reaction to the three issues raised. Their recommenda- 
tions were all taken into account in drafting this response, which we trust is both responsible 
• and comprehensive. * 

Although the Children's English and Services Study contract has been completed and the 

work accepted, LM&A has willingly accepted this opportunity to clarify concerns associated 

with that effort. Further, given previous staff commitments, we appreciate your extension of 

time to February 15, 1980, which has allowed us to prepare a more detailed response. 
* ♦ 

Sincerely, " ^ 

Lourdes Miranda , 
President 

L. Miranda and Associates, Inc. 
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Issue Were the items that were 

Number One: selected for inclusion in 
the Language Measure- 
ment and Assessment In- 
ventory (LM&AI) selected 
properly? 

The first issue is one that we have ad- 
dressed before and appears to be raised by 
individuals relatively new to the LM&AI, 
who may be unclear about the instrument's 
functional intent as well as unaware of its 
theoretical underpinnings. 

As stated in the Executive Summary (p iv) 
of the CESS Draft Report of September 6, 
/ 1979 (and, the later revision of November 
1979): "The test in English was designed to 
meet the definition of limited proficiency in 
the Bilingual Education Act." According to 
Title VII (Bilingual Education Programs) of 
the 1965 Elementary and Secondary Educa- 
tion Act — the legislation still in effect during 
the development of the LM&AI— individuals 
of limited English proficiency were those who 
had M . . . difficulty speaking and understand- 
ing instruction in the English language "[Sec. 
703(a) (1) (B)]. This is admittedly a narrow 
definition. It does not refer to the individual's 
proficiency in a native language nor to the 
relative balance between languages or domi- 
nance of one language over the other, it cfoes 
not even refer to general communicative 
competence in the EngHsh language. The 
crux of the definition is instruction in the 
English language. 

The reviewers rephrased this first issue of 
concern to ask: Is English language profi- 
ciency the dimension on which the scores 
vary, or are other dimensions associated with 
variation in the scores? Oar answer then is x 
yes, other dimensions are associated with 
score variation. However, these dimensions 
include skills that are also necessary to func- 
tion successfully at any given level in the U.S. 
public Educational system. The reviewers 
have suggested that the intermingling of 
items that measurfc memory and cognitive 
abilities in addition to language skills is inap- 
propriate. We maintain that language profi- 
ciency is inextricably related to memory and 
cognitive abilities. It is difficult to imagine 
how a test of "pure" linguistic competence 



could have been developed should we have 
been^asked to do so. Inasmuch as the pre- 
scribed function of the LM&AI was to meas- 
ure language proficiency as it relates to class- 
room instruction, it was essential for us to 
measure the ability to deal successfully with 
academic classroom tasks that are often as 
clearly reliant on memory and cognitive abili- 
ties as pn English language skills. 

We hope that the preceding comments will 
clarify any misunderstandings regarding the 
instrument's (1) functional intent, i.e., the 
development of an instrument for identifying 
individuals of limited English proficiency 
who are not able to learn successfully in a 
classroom where English is the language of 
instruction, and (2) theoretical underpin- 
♦ nings, e.g., that language proficiency and 
memory and cognitive skills are inextricably 
related. 

As the reviewers note, nhe LM&AI is a 
domain-referenced test. The issue of the 
appropriateness of item selection then be- 
comes twofold: Are the proper domains 
represented, and are items within a domain 
relevant to that domain? Our answer is an 
emphatic yes. The domains themselves, as 
well as the items within each domain, were 
^carefully selected and reviewed by dozens of 
educators, curriculum specialists, psycholo- 
gists, counselors, administrators, and evalua- 
tors. The expert panel members and their 
respective institutional affiliations are* listed 
in Appendix A, which also indicates the 
number of CESS Review Panel sessions each 
member attended. In addition, there were 
three field tests of the instrument, at which 
time about 100 teachers and psychologists 
t experienced first-hand contact with the con- 
tents of the tests. These professionals "pro- 
vided critical feedback to the test developers 
which was of great value in developing sub- 
sequent revisions of the test. Furthermore, 
remarkable foresight on the part of the CESS 
Review Panel is evidenced by the fact that the 
very four domains selected in 1977 to consti- 
tute the instrument include the two addi- 
tional skills mentioned in the 1978 amend- 
ments to the Elementary and ^Secondary 
Education Act. In the Title Vlf (Bilingual 
Education Programs) amendment . . . the de- 
finition of individuals of limited English pro- 
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ficiency has been expanded to include those 
who. w . . . have sufficient difficulty speaking, 
reading, writing, or understanding the Eng- 
lish language to deny them the opportunity 
to attain levels comparable to others at their 
appropriate age and grade levels" [Sec. 703 
(a) ( 1)(B)]. Item selection within the domains 
was a two-fold process. First, each potential 
test item was studied and weighed for its pre- 
sumed representativeness of a given domain 
and assessed by competent judges as to its 
relevance to the skill being measured. Second, 
selected items were retained in subsequent 
versions on the basis of their ability to differ- 
entiate between two criterion groups. Inde- 
pendent as well as collective judgments were 
used in the successive versions of the 10 age- 
level tests. 

The final subissue of the first issue brought 
up by the reviewers deals again with the ques- 
tion of dominance and the distinguishing 
characteristics of criterion groups. The quo- 
tation on page 8 from the November Draft 
Report (p. II-6) is misleading when taken out 
of context. It ifnplies that the fluent English 
speaking children forming criterion group B 
were also of language minority groups. Only 
i^this were the case would the subsequent 
reviewers* comments be relevant: "It could be 
argued that the two groups differed on native 
language as well as English language profi- 
ciency. ... A potential solution to this 
problem would be to equate the two criterion 
groups on proficiency in a non-English lan- 
guage" (p. 8). However, as stated in' the fol- 
lowing paragraph on p. 11-6 of the November 
DrafrReport; "The fluent English speaking 
group would be composed of students who 
were native English speakers. . . . There is, 
therefore, no danger that the test scores dif- 
ferentiating the criterion groups have a par- 
tial language dominance loading. The influ- 
ence of language dominance on respedjke 
language Competence and development in bi- 
lingual children is a subject that psycho- 
linguists are just beginning to address, gener- 
ally through descriptive case study research. 
There arc as yet no empirical research find- 
ings in this area upon which to draw in test 
development. Furthermore, until the legisla- 
tion governing bilingual education programs 
expands to include concern about the devel- 



opment of the native language of linguistic 
minorities, language dominance is a moot 
point. 

Issue Were the cut-off scores for 

Number Two:* the LM&AI, which were 
determined and used to 
classify children as either 
English proficient or of 
limited English proficiency, 
set properly? 

Subissue 0#e: 

Unfortunately, this issue arose because a 
necessary revisioniin the Technical Report 
excerpt cited on page 26 of this volume Was 
not made. Item 5 describes one type of dis- 
criminant function analysis considered, but 
not the procedure ultimately decided upon. 
Although a discriminant analysis using sub-, 
scores offered the most powerful use of the 
test to classify pupils, it was decided against 
for practical reasons. An easier, more practi- 
cal discriminant > analysis procedure was 
chpsen which use's the total score as a single 
scale and thus produces a single cut-off score. 
Item 5* should have been changed to read: 
"tJse discriminant function analysis (DFA) 
which utilizes total test scores to determine a 
centroid point, which can act as the critical 
point." Thus, the comments (1) and (2) on 
page 27 of this volume are no longer appli- 
cable as subscores were indeed Jiot used as 
the reviewers suspected. We apologize for 
this oversight on our part and the confusion 
it created. * 

■ * 

Sub issues 

Two and Three: 

Balancing the*misclassifications (as the 
mechanism suggested for revising cut-off 
scores would do) was an option considered 
by LM&A. We chose, instead, to minimize 
overall misclassifications, i.e., to improve the 
overall "hit-rate." The "Least Squares" ap- 
proach was selected in order to minimize the 
total number of errors. We are aware that } 
approach chosen results in relatively tonser- 
vative estimates of LEP children because it 
classifies more LEP children as FES than 
vice versa. While this has unfortunate impli- 
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cations for the individual child who will be 
deprived of participation in a needed ESL or 
bilingual program because he or she is mis- 
classified, the CESS concerns were with an 
overall estimate of numbers. The method 
proposed by the reviewers does not take into 
account the potential consequences of mis- 
classifying the FES group while correcting the 
LESAs. Increasing the number of FES pupils 
misclassified in a new "unbiased* count 
would touch upon a sensitive political issue. 
As you may be aware, since the inception of 
bilingual education in this country, the issue 
of providing equal educational opportunities 
te children of non-English language back- 
ground has been surrounded by controversy. 
The issue has centered around Congressional 
concern with tightening definitions of eligible 
children to ensure that resources reach those 
who are most in need. The AIR Evaluation 
Study of Title VII, by estimating that only 
approximately one-third of Title VII pupils 
were LEP, is an example of how the contro- 
versy has been fueled. 

L. Miranda and Associates, Inc., an 
Hispanfc-owned, minority firm, has been 
particularly' sensitive to this controversy. 
With that sensitivity in mind, the Contractor, 
with the concurrence of the NIE Project 
^Director, opted for the more- conservative 
approach which appeared to offer an objec-* 
tive- estimate of children of limited English 



proficiency beyond Congressional reproach. 

Another reason that an approach balanc- 
ing the misclassifications was not selected i^ 
that it would have failed to account for the 
unique leptokurtic curve of the FES group. 
Because of th^ leptokurtic distribution, the 
misclassifications were unbalanced. Since the 
imbalance is an artifact of the distribution, it 
makes no sense to balance the misclassifica- 
tions and thereby give an artificial impression 
of where cut-off scores ought to be. 



Issue 

Number Three: 



What were the effects of 
non-response bias on the 
counts and estimate of 
the number of LEP chil- 
dren? 



^ We do, ofcojjfse, realize that the potential 
effects of non-response bias are a reality. 
Unfortunately, we were working unfler time 
and cost restraints that did not permit us to 
carry out an' analytical inquiry into non- 
response bias, nor was that activity in the 
originally proposed study design agreed upon 
by NIE. 

Because of the way tHc CESS was con- 
ducted, it would not be difficult to go back to 
the computer for an empirical investigation 
of non-response bias without collecting addi- 
tional data. The concentric circle diagram fol- 
lowing serves^to illustrate the kind of data on 
tape. 



Diagram I: Response Rates of Identified NELB Households 





Area A- Households responding to Household Screcner but not to Household Questionnaire 
Arc# B Households responding to Household Screcner and .Questionnaire, but not to LM&AI 
Area C. Households, responding to Household Screcner. Household Questionnaire, and LM&AI 



. The total area (A, B, and C) represents the 
non-English language background (NELB) 
households responding to the Household 
Screener (76.199c of the total sample). This 
response rate is not particularly disturbing 
when closer analysis reveals that only 57 out 
of the 7,214 households not responding were 
expected to be language minority (November 
Draft Report, page 111-10). The actual 
response rate of NELB households within the 
total sample was very good indeed. Since the 
relevant population for the Household Ques- 
tionnaire and the LM&A1 test was NELB, 
the purpose of the Household Screener was 
to narrow the sample to that population, 
therefore, ttt^rp could be onl> minimal non- 
response bia^t this level. 

The restfonse rate for the Household 
Questionnaire was 93.15% nationally (Areas 
B and C), indicating that once a language 
minority household was located, cooperation 
was very high. Area A, then, represents the 
households that responded to the screener 
but did not respond to the Household Ques- 
tionnaire The national response rate for the 
language test was 84.58%, represented by the 
Area C. The largest categories within the 
non-response group to the LM&A1 test 
(Area B) were refusals by parents and 
unavailability of children at appointed test 
time (November Draft Report, page 111-12). 

Since Areas A and B represent the non- 
response groups to the Household Question- 
naire and the LM&A1 test respectively, the 
two groups could each be compared with the 
Area C group with regard to the household 
linguistic environment. The Area A group 
(those not responding to the Household 
Questionnaire) and the Area C group (the 
total response group) could be compared, for 
example, on their responses to items S2 - S5 
on the Household Screener Test (e.g., "What 
language do the people in this household usu- 
al!}! often! sometimes speak here at home?") 
Since the response rate for the ^pusehold 
Questionnaire was relatively high (93.75%), 



this comparison would not be as crucial as 
the one that could be made between the Area 
B group (non-respondents to the LM&A1 
test) and the Area C group (total response). 
Because of the lower response rate on the' 
LM&A1 (84.58%), this would appear to be 
the more relevant comparison for empiricafl> 
investigating differences between respondents 
and non-respondents. Whereas the previous 
comparison would be made witfTttie hojise- 
hold as the unit, this comparison would be 
individual child based, .comparing parent- 
reported linguistic behavior and ability of 
children who were not tested with those 
children who were tested. Several items on 
the Household Questionnaire lend them- 
selves to this end. These items include: 

4. How well does (Name) under- 
stand spoken English? 

5. How well does (Name) speak 

English? 

6. What language does (Name) 

usually speak to (Name's) 

brothers and sisters? 

7. What language does (Name) 

usually speak to (Name's) b esl 

friends? 

Thus, as we have shown, an empirical in- 
vestigation of non-response bias could be 
carried out without further d ^'"collection, 
but it would involve the cost of computer and 
staff analysis time. Since this activity wa$ not 
part of the approved CESS study design, wb 
do not feel obligated to carry out the activity 
at our own expense, nor do we feel th^t the 
issue is of such magnitude as to jeopardize 
the validity of our findings. It is an intriguing 
issue, however, and one which we would cer- 
tainty be interested in pursuing with addi- 
tional funding. This is probably a dimension 
which should be built into any further CESS 
study research design. 
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